IITM Math | EigenSpace

📐

1. Conceptual Overview

In machine learning and statistics, we often deal with high-dimensional data. PCA is a dimensionality reduction technique that transforms a large set of variables into a smaller one that still contains most of the information.

Geometrically, PCA finds a new coordinate system for the data such that the greatest variance by some scalar projection of the data comes to lie on the first coordinate (the first principal component), the second greatest variance on the second coordinate, and so on.

📈

2. The Covariance Matrix

Assume we have a centered data matrix $X$ of size $n \times p$ (where $n$ is the number of samples and $p$ is the number of features). The sample covariance matrix $S$ is defined as:

$$S = \frac{1}{n-1} X^T X$$

The covariance matrix is a symmetric, positive semi-definite matrix of size $p \times p$. Its diagonal entries represent the variance of individual features, while the off-diagonal entries represent the covariance between pairs of features.

CORE THEOREM

Spectral Theorem

CODE ZERO // EXAMPLE

Because the covariance matrix $S$ is real and symmetric ($S^T = S$), the Spectral Theorem guarantees that it can be diagonalized by an orthogonal matrix.

Specifically, there exists an orthogonal matrix $V$ (where $V^T V = I$) and a diagonal matrix $\Lambda$ such that:

$$S = V \Lambda V^T$$

The columns of $V$ are the eigenvectors (the principal directions), and the diagonal elements of $\Lambda$ are the corresponding eigenvalues (the magnitude of variance along those directions).

CODE ZERO // EXAMPLE

Given a simplified $2 \times 2$ covariance matrix $S$:

$$S = \begin{bmatrix} 2 & 1 \\ 1 & 2 \end{bmatrix}$$

Step 1: Find the eigenvalues ($\lambda$). Solve the characteristic equation $\det(S - \lambda I) = 0$:

$$ \begin{aligned} \det \begin{bmatrix} 2 - \lambda & 1 \\ 1 & 2 - \lambda \end{bmatrix} &= 0 \\ (2 - \lambda)^2 - 1 &= 0 \\ \lambda^2 - 4\lambda + 3 &= 0 \\ (\lambda - 3)(\lambda - 1) &= 0 \end{aligned} $$

The eigenvalues are $\lambda_1 = 3$ and $\lambda_2 = 1$. The first principal component corresponds to $\lambda_1 = 3$, meaning it captures 75% of the total variance ($\frac{3}{3+1}$).

💡 A-HA MOMENT

The eigenvectors point in the direction of the new axes, and the eigenvalues tell us the "stretch" or variance of the data along those new axes. You just learned the hardest part of PCA.